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Abstract 

The  Department  of  Defense  (DoD)  acquires  billions  of  dollars  of  supplies  and  services  every 
year.  In  fiscal  year  (FY)  2013,  the  DoD  obligated  over  $258  billion  for  military-unique  weapon 
systems  as  well  as  commercial  supplies  and  services.  An  integral  part  of  the  DoD’s  contract 
management  process  is  the  source  selection  phase  when  offerors’  proposals  are  evaluated 
and  the  contract  award  decision  is  made.  A  critical  aspect  of  the  source  selection  phase  is 
the  evaluation  of  contractor  past  performance  information  as  part  of  the  overall  proposal 
evaluation  process.  The  DoD  uses  the  Past  Performance  Information  Retrieval  System 
(PPIRS),  which  consists  of  contractor  report  cards  extracted  from  the  Contractor 
Performance  Assessment  Reporting  System  (CPARS).  In  this  research,  we  examine  the 
value  of  CPARS  report  card  narratives  for  service  contracts  as  they  relate  to  their  associated 
objective  scores.  Our  primary  focus  in  this  research  is  to  examine  if  the  CPARS  report  card 
written  narrative  section  provides  value  to  the  contractor  performance  evaluation  process. 
Our  data  analysis  includes  sentiment  and  statistical  analysis,  as  well  as  interviews  with 
government  agency  contracting  professionals.  Using  CPARS  data,  narrative  analyses,  and 
interviews,  we  answer  the  following  research  questions:  (1 )  To  what  degree  are  government 
contracting  professionals  submitting  to  CPARS  contractor  performance  narratives  in 
accordance  with  the  guidelines  provided  in  the  CPARS  user’s  manual?  (2)  What  is  the  added 
value  of  the  contractor  performance  narratives  beyond  the  value  of  the  objective  scores  for 
performance?  (3)  What  is  the  statistical  relationship  between  the  sentiment  contained  in  the 
narratives  and  the  objective  scores  for  contractor  evaluations?  The  research  revealed  that 
there  are  a  variety  of  opportunities  to  improve  the  contracting  process  specifically  related  to 
the  narrative  portion  of  past  performance  assessment  reports. 

Introduction 

The  Government  Accountability  Office  (GAO)  reported  that  the  poor  management  of 
service  contracts  has  undermined  the  government’s  ability  to  obtain  a  good  value  for  the 
money  spent  and  has  contributed  to  the  GAO’s  decision  to  designate  management  of 
services  contracts  as  a  high-risk  area  for  the  Department  of  Defense  (DoD;  GAO,  2013b).  In 
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fact,  as  stressed  in  a  recent  memorandum  for  acquisition  professionals  by  the  under 
secretary  of  defense  for  acquisition,  technology,  and  logistics  (US[AT&L]),  improving  the 
efficiency  of  acquisition  of  products  and  services  is  of  utmost  importance  to  the  DoD  (USD 
[AT&L],  2010.  More  specifically,  in  a  later  memorandum,  the  USD  (AT&L)  focused  on 
“improving  tradecraft  in  services  acquisition”  (USD  [AT&L],  2010,  p.  5)  by  strengthening  and 
improving  the  services  contracting  process.  An  important  part  of  the  services  acquisition 
process  is  the  evaluation  of  contractor  past  performance  information  using  the  Contractor 
Performance  Assessment  Reporting  System  (CPARS). 

The  CPARS  report  is  initiated  by  DoD  contracting  officers  during  the  contract 
closeout  phase  of  the  contract  management  process  for  documenting  contractor 
performance  information  on  the  completed  contract.  It  is  also  used  by  DoD  contracting 
officers  during  the  source  selection  phase  as  part  of  the  evaluation  of  contractor  proposals. 
The  CPARS  report  contains  contractor  performance  information  using  objective  scores  in 
five  categories:  Quality,  Schedule,  Cost  Control,  Business  Relations,  and  Management  of 
Key  Personnel.  In  addition  to  these  five  objective  categories,  the  CPARS  reports  also 
provide  a  subjective  narrative  section  where  the  contracting  officer  provides  a  descriptive 
narrative  of  the  contractor’s  performance. 

Although  the  use  of  contractor  past  performance  information  is  an  important  aspect 
of  the  DoD  contract  management  process,  the  GAO  has  identified  many  process 
deficiencies  in  the  documentation  and  management  of  CPARS  reports.  GAO  reports  have 
shown  that  DoD  agencies  do  not  always  complete  the  required  contractor  past  performance 
reports  (GAO,  2007,  2009b,  2013a,  2014).  The  2012  National  Defense  Authorization  Act 
(NDAA)  required  the  DoD  to  “develop  a  strategy  to  ensure  that  evaluations  in  past 
performance  databases  used  for  making  source  selection  decisions  are  complete,  timely, 
and  accurate”  (GAO,  2014,  p.  4).  Additionally,  the  2013  NDAA  required  a  “government-wide 
strategy  to  ensure  that  timely,  accurate,  and  complete  information  on  contractor 
performance  is  included  in  past  performance  databases  used  by  executive  agencies  for 
making  source  selection  decisions”  (GAO,  2014,  p.  1). 

Subsequently,  the  DoD  increased  focus  on  training  and  education  for  contracting 
officers,  which  resulted  in  an  increase  in  contractor  performance  assessments  being 
completed  and  submitted.  In  2013,  the  GAO  noted  significant  gains  in  CPARS  completion 
rates:  56%  of  required  reports  were  completed  in  201 1  while  74%  were  completed  in  2013. 
However,  according  to  the  same  GAO  report,  over  half  of  the  CPARS  reports  were 
submitted  late.  More  importantly,  many  CPARS  reports  contain  narratives  that  are  either 
insufficiently  detailed  or  are  in  conflict  with  their  associated  objective  scores.  Late  reports 
lacking  sufficient  accurate  information  provide  less-than-optimal  information  to  the 
contracting  professionals  that  rely  on  these  report  cards  for  source  selection  and  contract 
administration  purposes  (GAO,  2013a). 

The  purpose  of  this  research  is  to  determine  the  value  of  the  CPARS  narratives  in 
services  acquisition  by  comparing  the  relationships  between  the  subjective  narratives  and 
the  associated  objective  scores.  Our  analysis  allows  us  to  suggest  improvements  to  the 
CPARS  management  process,  thus  leading  to  greater  and  more  effective  utilization  of  the 
CPARS  reports  in  services  acquisition. 

Research  Methodology 

This  research  examines  the  value  of  CPARS  report  card  narratives  for  service 
contracts  as  they  relate  to  their  associated  objective  scores.  Our  primary  focus  in  this 
research  is  to  examine  if  the  CPARS  report  card  written  narrative  section  provides  value  to 
the  contractor  performance  evaluation  process.  Our  data  analysis  includes  sentiment  and 
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statistical  analysis,  as  well  as  interviews  with  government  agency  contracting  professionals. 
Using  CPARS  data  collected  by  graduate  students,  Wilhite,  Stover,  and  Hart  (2013),  and 
narrative  analyses  and  interviews  conducted  by  graduate  students  Black,  Henley,  and  Clute 
(2014),  we  answer  the  following  research  questions: 

1 .  What  is  the  added  value  of  the  contractor  performance  narratives  beyond  the 
value  of  the  objective  scores  for  performance? 

2.  What  is  the  statistical  relationship  between  the  sentiment  contained  in  the 
narratives  and  the  objective  scores  for  contractor  evaluations? 

3.  To  what  degree  do  the  interview  findings  contradict,  support,  or  enhance  the 
findings  for  our  research  questions? 

Literature  Review 

Federal  procurement  policy  requires  that  agencies  collect  information  regarding  a 
contractor’s  performance  under  previously  awarded  contracts  for  all  contracts  over  $100,000 
and  make  that  information  available  for  use  in  future  contract  award  decisions  (Nash  et  al., 
2007).  The  collection  of  contractor  performance  information  occurs  during  the  contract 
closeout  phase  using  the  DoD  CPARS  (Rendon  &  Snider,  2008). 

The  CPARS  assessment  data  reflects  the  contractor’s  performance  in  specific  areas 
including  quality,  schedule,  cost  control,  business  relations,  management  of  key  personnel, 
and  utilization  of  small  business.  The  “Quality”  rating  assesses  the  contractor’s  qualitative 
performance  and  compares  it  to  the  requirements  stated  in  the  contract.  The  “Schedule” 
rating  assesses  the  contractor’s  ability  to  meet  schedules  outlined  in  the  contract  such  as 
milestones,  task  orders,  delivery  schedules,  and  administrative  requirements.  The  “Cost 
Control”  rating  assesses  the  contractor’s  ability  to  forecast,  manage,  and  control  the  costs 
associated  with  performing  contracted  services.  The  “Business  Relations”  rating  assesses 
the  contractor’s  ability  to  coordinate  its  business  activities  such  as  cooperate  behavior, 
customer  satisfaction,  management,  and  attitude  towards  customers.  The  “Management  of 
Key  Personnel”  rating  assesses  the  contractor’s  ability  to  maintain  qualified  individuals  in 
key  positions  as  outlined  in  the  contract.  The  “Utilization  of  Small  Business”  rating  assesses 
the  contractor’s  ability  to  integrate  small  businesses  in  the  execution  of  the  contract  (Wilhite 
et  al.,  2013). 

The  CPARS  assessment  rates  the  contractor  in  these  areas  using  the  rating  scales 
Exceptional,  Very  Good,  Satisfactory,  Marginal,  and  Unsatisfactory.  It  should  be  noted  that 
the  contractor  is  allowed  to  review  the  CPARS  assessment  and  provide  comments  back  to 
the  government  assessing  official  prior  to  the  government  finalizing  the  CPARS  report. 

During  the  source  selection  phase  of  government  negotiated  procurement,  contractor 
performance  information  is  used  in  evaluating  offerors  and  in  making  a  contract  award 
decision  (Rendon  &  Snider,  2008).  In  this  phase,  the  government  agency  accesses  the 
contractor  performance  information  through  the  DoD  Past  Performance  Information 
Retrieval  System  Report  Cards  (PPIRS-RC)  database.  During  source  selection  in  the 
evaluation  of  offeror’s  proposals,  the  government  agency  uses  the  contractor  past 
performance  information  to  determine  if  the  offeror  meets  the  required  standards  of 
responsibility  as  stated  in  the  federal  procurement  policy,  and,  depending  on  the  basis  of  the 
award  stipulated  in  the  solicitation,  uses  the  contractor’s  past  performance  ratings  to  justify 
an  award  to  a  higher-priced  offeror. 

The  contractor  performance  information  reported  in  CPARS  and  accessible  through 
PPIRS  provides  outcome-based  data  that  can  be  used  to  identify  successful  contracts.  The 
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successful  contracts  determined  by  using  contractor  performance  information  have  been 
used  in  our  previous  research  to  identify  the  contract  variables  that  lead  to  contract  success. 

In  2014,  with  the  assistance  of  our  graduate  students  (Wilhite  et  al.,  2013),  we 
accessed  the  past  performance  database  to  collect  contractor  performance  ratings  on  715 
completed  Army  services  contracts  to  determine  if  the  contracts  were  successful  or  not 
successful.  Using  statistical  analysis,  we  investigated  whether  certain  contracting  variables 
such  as  type  of  service,  contract  dollar  value,  level  of  competition,  and  contract  type  affected 
the  success  of  the  contract.  The  detailed  results  of  our  analysis  are  presented  in  Rendon, 
Apte,  &  Dixon  (2014).  Our  research  findings  concluded  that  the  S  type  services  (Utilities  and 
Housekeeping)  had  the  highest  failure  rate  of  all  the  product  service  codes  analyzed.  We 
also  found  that  contracts  with  a  dollar  value  from  $50  million  to  $1  billion  had  the  highest 
failure  rate  of  all  the  contract  categories.  We  found  that  contracts  competed  competitively 
had  the  highest  failure  rate  when  compared  to  the  other  two  forms  of  competition  available. 
Furthermore,  we  found  that  contracts  structured  as  a  combination  contract  had  the  highest 
failure  rate  when  compared  to  the  other  five  types  of  available  contracts.  Finally,  the  results 
of  our  significance  testing  showed  that  Contractual  Amounts  and  Contract  Type  were  our 
only  statistically  significant  variables  (Wilhite  et  al.,  2013). 

Our  past  research  using  CPARS  data  identified  some  interesting  areas  worthy  of 
further  exploration.  These  areas  include  analyzing  the  narrative  portion  of  the  CPARS 
ratings  to  determine  alignment  with  the  objective  ratings,  as  well  as  the  value  added,  not 
only  in  the  narrative  portions,  but  also  in  the  usefulness  of  the  CPARS  as  a  contractor 
assessment  tool.  This  is  the  focus  of  our  current  research  project. 

Research  Design 

Our  research  examines  the  value  of  the  CPARS  report  card  narratives  for  service 
contracts  as  they  relate  to  their  associated  objective  scores.  The  primary  focus  in  this 
research  is  examining  if  the  CPARS  report  card  written  narrative  section  provides  value  to 
the  contractor  performance  evaluation  process.  Our  data  analysis  included  a  sentiment 
analysis  and  statistical  analysis,  as  well  as  interviews  with  government  agency  contracting 
professionals. 

With  the  assistance  of  our  most  recent  MBA  thesis  students  (Black,  Henley,  and 
Clute),  we  performed  a  sentiment  analysis  of  the  715  Army  service  contract  CPARS  report 
card  narratives  accessed  in  our  previous  research  (Rendon  et  al.,  2014).  Our  students  used 
the  CPARS  Quality  Checklist  as  a  basis  for  developing  the  criteria  for  the  categories  and 
values  for  the  sentiment  analysis  (CPARS  Best  Practices,  CPAR  Quality  Checklist,  n.d.).  In 
the  sentiment  analysis,  the  student  researchers  scored  each  narrative  along  the  dimensions 
of  quality,  robustness,  compliance  with  directions  in  the  CPARS  Quality  Checklist,  and  its 
value  and  content  compared  to  its  related  objective  scores  from  the  CPARS  report  cards. 
Independent  researchers’  scores  were  compared  across  a  small  sample  to  ensure  inter¬ 
rater  reliability. 

We  conducted  a  statistical  analysis  of  the  relationship  between  the  sentiment 
analysis  scores  and  their  associated  objective  rating  scores.  This  analysis  investigated 
correlating  relationships  between  the  sentiment  scores  and  the  objective  rating  scores  for 
the  same  CPARS  report.  Our  purpose  was  to  explore  the  relationships  between  the 
sentiment  scores  and  the  objective  rating  scores  to  reveal  the  extent  of  the  value  of  the 
narratives. 

Our  students  (Black  et  al.,  2014)  also  conducted  interviews  with  contracting 
professionals  from  two  DoD  contracting  agencies.  These  interviews  focused  on  the 
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agencies’  use  of  CPARS  and  other  sources  of  contractor  past  performance  information  and 
agencies’  value  of  the  CPARS  narratives  compared  to  the  objective  rating  scores. 

Findings  and  Analysis 

In  this  section,  we  present  an  analysis  of  our  findings.  The  primary  purpose  of  this 
research  was  to  determine  the  value  of  the  CPARS  narratives  in  services  acquisition  by 
comparing  the  relationships  between  the  subjective  narratives  and  the  objective  scores.  We 
first  present  the  findings  of  the  sentiment  and  statistical  analysis  by  focusing  on  each  of  the 
criteria  used  in  the  analysis. 

1 .  Do  the  narratives  address  all  performance  areas  assessed?  Overall,  the 
narratives  address  all  performance  areas  assessed  -82%  of  the  time.  This  is 
less  problematic  with  unsuccessful  contracts  at  -95%  than  with  successful 
contracts  at  -81  %.  The  difference  in  the  proportion  of  times  that  the  narrative 
addresses  all  performance  areas  assessed  in  successful  and  unsuccessful 
contracts  is  statistically  significant  (p  <  .05;  Black  et  al.,  2014,  p.  41). 

2.  Are  narratives  based  on  objective  data?  Overall,  the  narratives  are  based  on 
objective  data  -77%  of  the  time.  However,  in  unsuccessful  contracts,  the 
narratives  are  based  on  objective  data  100%  of  the  time.  This  is  significantly 
different  from  the  -77%  in  successful  contracts  (p  <  .01;  Black  et  al.,  2014,  p. 
41). 

3.  Are  narratives  free  of  statements  to  avoid?  Overall,  the  narratives  are  free  of 
statements  to  avoid  -97%  of  the  time.  This  is  slightly  more  problematic  with 
unsuccessful  contracts  at  -86%  than  with  successful  contracts  at  -97%  (p  < 
.01;  Black  etal.,  2014,  p.  41). 

4.  Are  narratives  robust  and  comprehensive?  Overall,  the  narratives  are  robust 
and  comprehensive  -63%  of  the  time.  This  is  less  problematic  with 
unsuccessful  contracts  at  -91%  than  with  successful  contracts  at  -62%  (p  < 
.01;  Black  etal.,  2014,  p.  41). 

5.  Could  a  layman  understand  the  description  of  the  work  performed?  Overall, 
the  narratives  are  written  so  that  a  contracting  layman  should  understand  the 
work  performed  -64%  of  the  time.  This  is  less  problematic  with  unsuccessful 
contracts  at  -82%  than  it  is  with  successful  contracts  at  -64%  (p  <  .05;  Black 
etal.,  2014,  p.  41). 

6.  Is  the  narrative  beneficial  above  and  beyond  objective  scores?  Using  a  Chi 
Square  Test,  we  determined  that  there  was  a  difference  between  successful 
and  unsuccessful  contracts  in  whether  the  narratives  were  beneficial  above 
and  beyond  the  objective  scores.  Unsuccessful  contracts  tended  to  have 
more  beneficial  CPARS  report  card  narratives  than  successful  contracts 
(Black  et  al.,  p.  42).  Overall,  the  narrative  provides  an  unsatisfactory  amount 
of  beneficial  data  to  the  user  -12%  of  the  time.  However,  there  were  no 
unsuccessful  contracts  that  provided  an  unsatisfactory  amount  of  beneficial 
data.  The  narrative  provides  a  marginal  amount  of  beneficial  data  -22%  of 
the  time.  There  were  no  unsuccessful  contracts  that  provided  a  marginal 
amount  of  beneficial  data.  The  narrative  provides  a  satisfactory  amount  of 
beneficial  data  -28%  of  the  time.  The  narrative  provides  a  very  good  amount 
of  beneficial  data  -21%  of  the  time.  The  narrative  provides  an  exceptional 
amount  of  beneficial  data  -18%  of  the  time.  This  is  much  more  likely  to  occur 
with  unsuccessful  contracts  than  with  successful  contracts  at  -17%  (Black  et 
al.,  2014,  p.  42). 
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7.  Do  the  narratives  correlate  to  the  objective  scores  assigned?  Using  the  Chi 
Square  Test,  we  determined  that  there  was  not  a  difference  between 
successful  and  unsuccessful  contracts  in  whether  the  narrative  correlates  to 
the  objective  scores  assigned.  Overall,  the  narrative  sentiment  is 
contradictory  to  more  than  one  of  the  objective  scores  assigned  ~2%  of  the 
time.  The  narrative  sentiment  is  contradictory  to  one  of  the  objective  scores 
assigned  ~6%  of  the  time.  The  narrative  sentiment  is  satisfactory  in 
describing  accurately  why  the  objective  scores  are  assigned  as  they  are 
-28%  of  the  time.  The  narrative  sentiment  is  very  successful  in  describing 
accurately  why  the  objective  scores  are  assigned  as  they  are  -40%  of  the 
time.  The  narrative  sentiment  is  exceptionally  successful  in  describing 
accurately  why  the  objective  scores  are  assigned  as  they  are  -24%  of  the 
time  (Black  et  al. ,  2014,  p.  42).  Figure  1  summarizes  the  results  of  the 
statistical  analysis. 
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Result 

Overall 

Unsuccessful 

Contract 

Successful 

Contract 

Successful  vs 

Unsuccessful 

Statistically 

Significant? 

P  Value 

Narrative 

Addresses  All 

Performance 

Areas  Assessed? 

No 

18.46% 

4.55% 

18.90% 

Yes 

0.043 

Yes 

81.54% 

95.45% 

81.10% 

Narrative  Based 

On  Objective 
Data? 

No 

22.52% 

0.00% 

23.23% 

Yes 

<.01 

Yes 

77.48% 

100.00% 

76.77% 

Nanath/e  Is  Free 

Of  Statements  To 

Avoid? 

No 

3.08% 

13.64% 

2.74% 

Yes 

<.01 

Yes 

96.92% 

86.36% 

97.26% 

Narrative  Is 

Robust  & 
Comprehensive? 

No 

37.48% 

9.09% 

38.38% 

Yes 

<01 

Yes 

62.52% 

90.91% 

61.62% 

Could  A  Layman 

Understand  Work 

Performed? 

No 

35.52% 

18.18% 

36.08% 

Yes 

0.042 

Yes 

64.48% 

81.82% 

63.92% 

Is  The  Narrative 

Score  1 

11.61% 

0.00% 

12.12% 

Score  2 

21.68% 

0.00% 

22.37% 

&  Beyond 
Objective  Scores? 

Score  3 

27.97% 

18.18% 

28.28% 

Yes 

<.01 

Score  4 

20.84% 

27.27% 

20.63% 

Score  5 

17.76% 

54.55% 

16.59% 

Does  The 

Narrative 

Correlate  To 
Objective  Scores 
Assigned? 

Score  1 

1.82% 

4.55% 

1.88% 

No 

0.141 

Score  2 

6.43% 

4.55% 

6.49% 

Score  3 

27.83% 

9.09% 

28.43% 

Scored 

40.42% 

40.91% 

40.40% 

Score  5 

23.36% 

40.91% 

22.80% 

Figure  1.  Results  of  Statistical  Database  Analysis 

(Black  et  al.,  2014) 

As  previously  discussed,  our  students  also  conducted  interviews  with  contracting 
professionals  from  two  DoD  contracting  agencies  (Black  et  al.,  2014).  These  interviews 
focused  on  the  agencies’  use  of  CPARS  and  other  sources  of  contractor  past  performance 
information  as  well  as  these  agencies’  value  of  the  CPARS  narratives  compared  to  the 
objective  rating  scores  in  the  source  selection  process.  The  findings  of  these  interviews  are 
summarized  as  follows: 

1 .  CPARS  is  still  often  not  reliable,  robust,  or  comprehensive  enough.  This 
results  in  source  selection  officials  not  placing  a  significant  amount  of  weight 
on  the  past  performance  evaluation  criteria  (Black  et  al.,  2014,  p.  44). 
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2.  Unsuccessful  contracts  tend  to  have  more  reliable,  robust,  and 
comprehensive  past  performance  information  available  in  their 
CPARS/PPIRS  reports  (Black  et  al. ,  2014,  p.  45). 

3.  The  appropriate  amount  of  weight  that  should  be  assigned  to  the  past 
performance  evaluation  criteria  in  making  a  source  selection  decision  should 
be  correlated  to  the  source,  availability,  quality,  and  relevancy  of  the  past 
performance  information  (Black  et  al.,  2014,  p.  45). 

4.  The  information  found  in  PPIRS  sometimes  contains  information  in  the 
narrative  that  is  either  contradictory  or  does  not  quite  match  up  with  the 
objective  scores.  When  the  objective  scores  and  narrative  sentiment  in 
PPIRS  is  mismatched,  contracting  professionals  tend  to  give  more  weight  to 
the  narrative  versus  the  objective  scores  (Black  et  al.,  2014,  p.  46). 

5.  Contracting  professionals  are  not  always  applying  due  diligence  in  identifying 
the  appropriate  contractor  entity  (e.g.,  CAGE  Code  or  DUNS  number)  in  the 
CPARS  reports.  This  is  resulting  in  contractor  past  performance  information 
not  being  fully  accessible  in  PPIRS  (Black  et  al.,  2014,  p.  46). 

6.  There  is  a  lack  of  reliable,  robust,  and  comprehensive  amount  of  past 
performance  information  available  in  PPIRS.  This  results  in  source  selection 
officials  soliciting  contractors  for  references  or  asking  contractors  to  fill  out  a 
past  performance  questionnaire  (Black  et  al.,  2014,  p.  47). 

7.  The  results  of  the  interviews  also  identified  recommendations  for  improving 
the  quality  of  CPARS  reports,  incorporating  data  analytics  tools  into  the 
PPIRS  database,  enhancing  the  monitoring  of  Contracting  Officer 
Representative  (COR)  workload,  improving  acquisition  workforce  training  on 
developing  CPARS  narratives,  and  improving  the  disclosure  of  CPARS 
program  office  audit  results  (Black  et  al.,  2014,  pp.  48-49). 

Summary,  Conclusions,  and  Recommendations 
Summary 

The  DoD  acquires  billions  of  dollars  of  supplies  and  services  every  year.  In  FY  201 3, 
the  DoD  obligated  over  $258  billion  for  military-unique  weapon  systems  as  well  as 
commercial  supplies  and  services  (USA  Spending,  201 3).  An  integral  part  of  the  DoD’s 
contract  management  process  is  the  source  selection  phase  when  offerors’  proposals  are 
evaluated  and  the  contract  award  decision  is  made.  A  critical  aspect  of  the  source  selection 
phase  is  the  evaluation  of  contractor  past  performance  information  as  part  of  the  overall 
proposal  evaluation  process.  The  DoD  uses  the  Past  Performance  Information  Retrieval 
Systems  (PPIRS),  which  consists  of  contractor  report  cards  extracted  from  the  Contractor 
Performance  Assessment  Reporting  System  (CPARS).  Although  the  use  of  contractor  past 
performance  information  is  an  important  aspect  of  the  DoD  contract  management  process, 
the  GAO  has  identified  many  process  deficiencies  in  the  documentation  and  management  of 
CPARS  reports.  GAO  reports  have  shown  that  DoD  agencies  do  not  always  complete  the 
required  contractor  past  performance  reports  (GAO,  2007,  2009b,  2013a,  2014).  More 
importantly,  many  CPARS  reports  contain  narratives  that  are  either  insufficiently  detailed  or 
conflict  with  their  associated  objective  scores.  Late  reports  lacking  sufficient  accurate 
information  provide  less-than-optimal  information  to  the  contracting  professionals  that  rely 
on  these  report  cards  for  source  selection  decisions. 

The  purpose  of  this  research  was  to  determine  the  value  of  the  CPARS  narratives  in 
services  acquisition  by  comparing  the  relationships  between  the  subjective  narratives  and 
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the  objective  scores.  Our  primary  focus  in  this  research  was  examining  if  the  CPARS  report 
card  written  narrative  section  provides  value  to  the  contractor  performance  evaluation 
process.  Our  data  analysis  included  sentiment  and  statistical  analysis,  as  well  as  interviews 
with  government  agency  contracting  professionals. 

Conclusions 

Using  CPARS  data  collected  by  graduate  students  Hart,  Stover,  and  Wilhite,  (2013), 
and  narrative  analyses  and  interviews  conducted  by  graduate  students  Black,  Henley,  and 
Clute  (2014),  we  answered  the  following  research  questions: 


1 .  What  is  the  added  value  of  the  contractor  performance  narratives  beyond  the 
value  of  the  objective  scores  for  performance?  Contracting  professionals  are 
doing  a  better  job  at  providing  beneficial  CPARS  data  in  the  narrative  when 
the  contract  is  unsuccessful  versus  when  it  is  successful.  Only  38.6%  of  the 
observed  CPARS  narratives,  whether  successful  or  unsuccessful,  provided  a 
very  good  or  exceptional  amount  of  beneficial  data  above  and  beyond  what 
could  be  gleaned  from  looking  over  the  objective  scores  assigned  (Black  et 
al.,  2014,  p.  51). 

2.  What  is  the  statistical  relationship  between  the  sentiment  contained  in  the 
narratives  and  the  objective  scores  for  contractor  evaluations?  Contracting 
professionals  are  developing  CPARS  narratives  that  contradict  at  least  one  of 
the  objective  scores  assigned  -8.3%  of  the  time.  Contracting  professionals 
were  slightly  better  at  matching  the  narrative  sentiment  to  the  objective 
scores  in  unsuccessful  contracts  (-81.8%  scoring  either  very  good  or 
exceptional)  than  in  successful  contracts  (-63.2%  scoring  either  very  good  or 
exceptional;  Black  et  al.,  2014,  p.  51). 

3.  To  what  degree  do  the  interview  findings  contradict,  support,  or  enhance  the 
findings  for  our  research  questions?  The  results  of  the  interviews  found  that 
the  CPARS  database  is  still  often  not  reliable,  robust,  or  comprehensive 
enough.  We  also  found  that  unsuccessful  contracts  tend  to  have  more 
reliable,  robust,  and  comprehensive  past  performance  information  available 
in  their  CPARS/PPIRS  reports.  Additionally,  the  appropriate  amount  of  weight 
that  should  be  assigned  to  the  past  performance  evaluation  criteria  in  making 
a  source  selection  decision  should  be  correlated  to  the  source,  availability, 
quality,  and  relevancy  of  the  past  performance  information.  Our  interviewees 
also  stated  that  the  information  found  in  the  PPIRS  database  sometimes 
contains  information  in  the  narrative  that  is  either  contradictory  or  does  not 
quite  match  up  with  the  objective  scores.  We  also  found  that  contracting 
professionals  are  not  always  applying  due  diligence  in  identifying  the 
appropriate  contractor  entity  in  the  CPARS  reports,  which  is  resulting  in  a 
lack  of  reliable,  robust,  and  comprehensive  amount  of  past  performance 
information  available  in  PPIRS.  Finally,  the  interview  results  also  identified 
recommendations  for  improving  the  quality  of  CPARS  reports,  incorporating 
data  analytics  tools  into  the  PPIRS  database,  enhancing  the  monitoring  of 
COR  workload,  improving  acquisition  workforce  training  on  developing 
CPARS  narratives,  and  improving  the  disclosure  of  CPARS  program  office 
audit  results  (Black  et  al.,  2014,  pp.  44-49). 


Recommendations 

Based  on  our  conclusions,  we  identified  the  following  five  recommendations: 
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Recommendation  1:  Training.  Training  should  be  implemented  for  all  services 
acquisition  members  that  interact  with  the  CPARS  and  PPIRS  databases.  Training  should 
focus  on  developing  comprehensive  narratives  ensuring  that  acquisition  team  members  can 
fully  understand  the  work  performed,  address  all  performance  areas  assessed  in  their 
objective  scores,  and  ensure  the  narratives  are  based  on  objective  data  (Black  et  al. ,  2014, 
pp.  54-55). 

Recommendation  2:  Process  Improvement.  The  DoD  needs  to  improve  the  quality 
of  past  performance  report  submissions  in  CPARS  and  PPIRS,  improving  the  source, 
availability,  quality,  relevancy,  and  accuracy  of  the  past  performance  information.  This  will 
allow  acquisition  teams  to  assign  higher  weights  to  past  performance  evaluation  criteria  in 
source  selection  decisions  (Black  et  al.,  2014,  pp.  54-55). 

Recommendation  3:  Data  Analytics.  Additional  data  analysis  tools  should  be 
incorporated  into  the  CPARS  and  PPIRS  database  to  better  assist  contracting  professionals 
in  identifying  past  performance  trends  for  a  particular  contractor  or  specific  type  of  service 
(Black  et  al.,  2014,  pp.  54-55). 

Recommendation  4:  Customer  Feedback.  The  CPARS  process  should  include 
customer  feedback  on  contractor  performance.  Currently,  only  the  acquisition  team  provides 
input  to  the  CPARS  report  card.  Customer  input  into  CPARS  will  encourage  the  submission 
of  more  accurate  and  robust  CPARS  report  cards  (Black  et  al.,  2014,  pp.  54-55). 

Recommendation  5:  COR  Manning  Levels.  Contracting  Officer  Representative 
(COR)  manning  levels  should  be  reviewed  throughout  the  DoD  to  ensure  that  organizations 
have  sufficiently  filled  COR  billets  to  manage  the  CPARS  process  (Black  et  al.,  2014,  pp. 
54-55). 
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